Data storage and file transfer

Data management on the DGX

Since the dgx is a machine shared by several users, the available memory space is de facto shared. For the moment, there are no restrictions per user, but this may change as storage usage evolves. So remember to monitor the amount of data you generate, bearing in mind that memory space is not infinite. A command you can use to do such monitoring is:

du -ahd0

to get the total space used by the current directory, with

-a, --all
    write counts for all files, not just directories

-d, --max-depth=N
    print the total for a directory (or file, with --all) only if it
    is N or fewer levels below the command line argument

-h, --human-readable
    print sizes in human readable format (e.g., 1K 234M 2G)

A given user storage is their home directory on the Data strorage, which is a 7.68TB drive.

Data transfer

Shell commands scp and rsync on Linux/Mac

In order to transfer files from the DGX or to the DGX, you can use the shell command scp from your computer.

The following command copies the directory my_dir on the home directory of the user username.

user@mycomputer:~$ ls my_dir/
file01.txt  file02.txt
user@mycomputer:~$ scp -r my_dir username@hubia-dgx.centralesupelec.fr:~/
file01.txt                                                                       100%  132KB 132.3KB/s   00:00
file02.txt                                                                       100%  132KB 132.3KB/s   00:00
user@mycomputer:~$

The following command makes a local copy of the directory my_dir from the homedir of the user username.

user@mycomputer:~$ scp -r username@hubia-dgx.centralesupelec.fr:~/my_dir .

To transfer bigger files (when the transfer is long enough), you should use the shell command rsync with the options --partial --progress. The --partial option allows rsync to keep partially transferred files and thus, in case an error occurs, to restart the tranfer from the partial files and not from scratch. The --progress tells rsync to print information showing the progress of the transfer. The following command copies the local directory my_dir on the workdir of the user username on the DGX.

user@mycomputer:~$ ls my_dir/
file01.txt  file02.txt
user@mycomputer:~$ rsync --partial -r my_dir username@hubia-dgx.centralesupelec.fr:~
sending incremental file list
my_dir/
my_dir/file01.txt
 10,737,418,240 100%  121.84MB/s    0:01:24 (xfr#1, to-chk=1/3)
my_dir/file02.txt
 10,737,418,240 100%  111.19MB/s    0:01:32 (xfr#2, to-chk=0/3)
user@mycomputer:~$

Using a scp client on Windows (WinSCP)

To transfer data from a Windows machine, a SCP client is mandatory. For example, WinSCP or FileZilla.